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Abstract 

In this paper we consider the problem of approximating a function be¬ 
longing to some function space $ by a linear combination of n trans¬ 
lates of a given function G. Using a lemma by Jones (1990) and Barron 
(1991) we show that it is possible to define function spaces and func¬ 
tions G for which the rate of convergence to zero of the error is 
in any number of dimensions. The apparent avoidance of the “curse of 
dimensionality” is due to the fact that these function spaces are more 
and more constrained as the dimension increases. Examples include 
spaces of the Sobolev type, in which the number of weak derivatives is 
required to be larger than the number of dimensions. We give results 
both for approximation in the L 2 norm and in the norm. The 
interesting feature of these results is that, thanks to the constructive 
nature of Jones’ and Barron’s lemma, an iterative procedure is defined 
that can achieve this rate. 
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1 Introduction 


Let $ be a normed space of functions and let A be a subset of $. The 
prototypical problem in approximation theory consists in approximating an 
element / of $ by an element of A, that is looking for an element in A that 
has minimum distance from /. It is also natural to consider the distance of 
f from A as 


h(/,A) = inf ||/- a|| (1) 

a£A 

and to study this quantity for different choices of A and / £ $. In the classical 
theory of approximation the set A is usually a linear ^-dimensional subspace 
Ak C $ (Lorentz, 1986) (the algebraic or the trigonometric polynomials of 
given degree and the splines with fixed knots are typical examples of such 
subspaces), while in nonlinear approximation theory the linear subspace Ak 
is replaced by a ^-dimensional manifold Mk (DeVore, 1991). Usually one has 
a family of manifolds {Mk}fL t such that Ufc Mk is dense in $ and 

Mi C M 2 C ... C M n C ... 

so that the quantity 8(f, Mk) is a monotone decreasing function of k converg¬ 
ing to zero and the approximation in Mk gets arbitrarily close to / provided 
one takes k sufficiently large. However, since the computational time needed 
to fold an approximation to / in Mk is going to increase with k, it is of great 
interest to know the rate of convergence to zero of 8(f,Mk) as a function 
of k. This rate of convergence can be taken as a measure of the complexity 
of / with respect to the manifolds Mk, in the sense that “simple” functions 
should have a fast rate of convergence. 

As an example, let us consider as space $ the space A d a of the functions 
whose partial derivatives of order s are bounded in the uniform norm on 
the d-dimensional cube / = [0, l\ d and satisfy a Lipschiz condition with 
exponent a (Lorentz, 1986, p. 50). On the space $ we consider the uniform 
norm ||/|| = max/|/(x)|. Choosing as manifold Mk the set of polynomials 
of degree n — 1 in each of the d variables, that is a linear space of dimension 
k = n d , the following bound can be obtained (Lorentz, 1986): 

s T Q: 

8(f,M k )< Ndk~— (2) 

where N is a constant that depends on / and s. 

From this example we see that the rate of convergence dramatically 
slows down when the dimension d increases, revealing the discouraging phe¬ 
nomenon known under the name of “curse of dimensionality” (Bellman, 
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1961). However, for every fixed number of dimensions, arbitrary inverse- 
power rates of convergence can be obtained if the smoothness index s is 
chosen big enough. This result is typical in linear approximation theory 
since the computation of the n-width of the space A d a shows that the best 
linear technique cannot improve the rate of convergence 0(k ~) (Lorentz 

1986, p. 135). 

Similar results, in both linear and nonlinear approximation theory (De- 
Vore, 1991), hold for other spaces of functions in which smoothness is mea¬ 
sured in a different way. We are therefore led to argue that in practical 
situations we can only approximate functions whose smoothness increases 
with the dimension. As an example we consider again the spaces A d sa for 
s = d. It is clear from eq. (2) that in this case the rate of convergence of 
polynomial approximation to an / £ K d da is 0(k~ 1 ) and it is in this sense 
“independent on dimensionality”. 

In a recent paper (1990) Jones showed how to construct a sequence of 
functions f n that approximate certain functions in a Hilbert space with a 
rate of convergence 0(-^=). A statement of Jones’ lemma is given in section 
2. An application of this result to projection pursuit regression and neural 
networks has already been presented in (Jones 1990; Barron 1991), where 
appropriate approximation schemes and spaces of functions in R d are 
described in which the complexity of approximation increases mildly with 
d. It is worthwhile to observe that this is obtained at the expense that the 
functions contained in are more and more “regular” when d increases. 
Moreover, it is not completely clear yet how computationally expensive the 
approximation f n may be. A very short review of Jones’ and Barron’s results 
is given in section 5. 

The aim of this paper is to present an application of Jones’ lemma to 
the approximation by linear combination of translates of a given function 
G. In particular for appropriate choices of G we obtain estimates for the 
rate of convergence of certain Radial Basis Functions schemes (Micchelli, 
1986; Powell, 1987; Dyn, 1991; Poggio and Girosi, 1990) on certain spaces 
of functions of Sobolev type. For the convenience of the reader we collect in 
the appendix a few known results about Sobolev spaces and integration of 
Banach valued functions. 


2 The Maurey-Jones-Barron Lemma 

Our result is based on a lemma by Jones (1990) on the convergence rate of an 
iterative approximation scheme in Hilbert spaces. A formally similar lemma, 
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brought to our attention by R. Dudley (Dudley, 1991), is due to Maurey, 
and was published by Pisier in 1981. However Jones’ lemma is constructive 
while Maurey’s is not. Here we report a version of the lemma due to Barron 
(Barron 1991) that contains a slight refinement of Jones’ result: 

Lemma 2.1 (Maurey-Jones-Barron) If f is in the closure of the convex 
hull of a set Q in a Hilbert space H with ||g|| < b for each g £ Q, then for 
every n > 1 and for c > b 2 — \\f\\ 2 there is a f n in the convex hull of n points 
in Q such that 


II f~fn 


C 


< - . 


n 


The interesting feature of this lemma is that the sequence {/n}^T 0 has the 
following structure: 


fn-\- 1 ^nfn T (1 (3) 

where a n and g n are chosen in order to “approximately solve” the following 
minimization problem: 


inf „ 11/- a nfn ~ (1 - a n )g n \\ 

a n eR,g n e c, 

where by “approximately solve” we mean that it is sufficient at each step to 
reach a distance from the inhmum of order The lemma is therefore 

constructive, providing a procedure that can achieve the prescribed rate. 

In order to exploit this result we need to define suitable classes of functions 
which are the closure of the convex hull of some subset Q of a Hilbert space 
H. We are therefore naturally led to study functions that can be represented 
as “infinite” convex combinations of the type 

oo oo 

f = Y1 a i > o , 9i £ Q , a i = 1 • ( 4 ) 

8=1 8=1 

One way to approach the problem consists in utilizing the integral represen¬ 
tation of functions. Suppose that the functions in a Hilbert space H can be 
represented by the integral 


/( x ) = / G' t (x)da(t) (5) 

J M 

where da is some measure on the parameter set Ai . If da is a finite measure, 
the integral (5) can be seen as an infinite convex combination of the type of 
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eq. (4), and therefore the function / belongs to the closure of the convex 
hull of some subset of H. In the next section we formalize this idea in the 
special case in which the functions G*t(x) are the translates G(x. — t) of a 
fixed function G and we show how it leads to define approximation techniques 
whose rate of convergence in appropriate spaces of functions is 0(-j^). 


3 Approximation by Translates of a Func¬ 
tion G 

Let G be a fixed function belonging to L 2 (R d ) = L 2 . We define the space Lg 
as the set of the functions of the form 

f = G* A (6) 

where A is any signed Radon measure whose total variation |A|#d = ||A|| is 
finite and the symbol * stands for the convolution operation. Assuming from 
now on that ||Cr||i, 2 = 1, the following inequality holds (Stein and Weiss, 
1971) 


showing the inclusion Lq C L 2 . It is natural to approximate elements of Lq 
by elements of the set 

n 

G n = {/ £ L 2 | / = ^2 Aj-Crti , A; £ i? , ti £ R d } , (7) 

8 = 1 

where we indicate by Gt the function G translated by the vector t, that is 
Crt(x) = Cr(x — t). Using lemma 2.1 we can now prove the following 

Theorem 3.1 Let f be a function in Lg, so that f = G * where G £ L 2 , 
¥A\l 2 = l, and A is a Radon signed measure of bounded total variation ||A||. 
Then f belongs to the L 2 -closure of the convex hull of the set 

A = {sG t | t g R d , | 5 | < ||A||} 

and there exist n coefficients c a and n vectors t a such that: 

n c 

||/- ]Tc„G'(x-t„)||/ 2 < - 

a =1 

for all c > ||A|| 2 - ||/||| 2 . 
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Proof: We consider the vector-valued function 


T : R d -> L 2 (R d ) 

such that 


T(t) = G t • 

The function T is continuous, hence A-measurable, moreover one has 

/ ||T(t)|MA|(t) = ||G|| i2 / d|A|(t) = ||A|| < +^> . 

JR d JR d 

Therefore it exists the Bochner integral of T with respect to A (see ap¬ 
pendix A): 


V = f T(t)d\(t) , 

JR d 

and by lemma (A.2) we have 

r/ £ co A (8) 

where A = {sGt | t £ R d , |s| < ||A||}, co A stands for the convex hull of 
the set A and the bar stands for the closure in L 2 . Now we shall prove that 
T] = f. This can be done by proving that 

F*f = F*t] , \/F* £ (L 2 )* (9) 

where (L 2 )* is the dual space of L 2} that is L 2 itself. From the properties of 
the Bochner integral we have: 

F*r, = F* [ B T(t)d\(t) = [ (F*Gt))d\(t) . 

J R d J R d 

Taking this into account, the identity (9) can be written as: 

[ dx </>(x) [ G(x —t)dA(t) = [ d\(t) [ dx </>(x)G(x — t) , \/cf> £ L 2 . 

J R d JR d J R d J R d 

Now by Fubini’s theorem the two sides of this last equation are equal, and 
therefore r] = /. 

By eq. (8) / = r] belongs to the L 2 closure of the convex hull of the set A, 
which is contained in the ball of radius ||A||. By the Maurey-Jones-Barron 
lemma we can find a set of n coefficients c a and n vectors t a such that: 
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71 c 

\\f - C aG(x - t a )\\l 2 < - 

a = 1 

for all c > C(f) = ||A|| 2 — ||/||/ 2 . □ 

In theorem (3.1) the approximation error is measured in the L 2 norm. Im¬ 
posing some restrictions on the function G a similar estimate can be obtained 
for other norms, and in particular for the L^ norm. In fact, suppose that 
G £ H s ’ 2 , where H s,2 (R d ) = H s ’ 2 is the Sobolev space of the functions whose 
weak derivatives up to order s are in L 2 (see Appendix B). Then one can 
easily see that theorem (3.1) can be formulated in the Hilbert space H s,2 
instead of L 2 : 

Theorem 3.2 Let f be a function such that f = G * where G £ i7 s,2 ; 
\\G\\h^ = 1, and A is a Radon signed measure of bounded total variation 
||A||. Then f belongs to the H s ’ 2 -closure of the convex hull of the set 

A = {sG t | t g R\ M < ||A||} 

and there exist n coefficients c a and n vectors t a such that: 

n c 

||/- J2 C »G(x-t a )\\ 2 H s,2 < ~ 

, n 

a=l 

for all c > ||A|| 2 - \\f\\ 2 H s, 2 - 

We notice that if the condition s > | holds, then the Sobolev embedding 
theorem (see Appendix B) guarantees that H s,2 C C° and that it exists 
Ci > 0 such that 


II • 11 oo < Cl || • \\ h s - 2 ■ 

Therefore the approximating sequence {f n } converges uniformly, and the 
following corollary holds: 

Corollary 3.1 Under the conditions of theorem (3.2), if s > | there exists 
n coefficients c a , n vectors t a and a constant ci such that: 

||/- ^c„G'(x-t„)||| oo < c^ 

a = 1 

for all c > ||A|| 2 - ||/||^, 2 . 
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From a practical point of view, in many cases, what it is really interesting 
is an estimate of the error in the sup norm, instead of the L 2 or H s,2 norm. 
Think for example of the problem of approximating the trajectory of a robot 
arm: it is clear that what is really needed in this case is a small L^ norm of 
the difference between the desired and the approximated trajectory, while a 
small L 2 norm is of little interest. 

Remark: we notice that the elements of the set G n defined by eq. (7) can 
also be seen as points of a manifold Mk whose dimension is k = n(d + 1). 
Therefore theorem (3.1) can also be formulated in terms of the number of 
parameters k that are needed to achieve a certain error, saying that if / £ Lq 
then 


<5(/.A4)<C'(/)^±l. 

If we compare this result with the typical estimates (DeVore, 1991), we 
notice that in this case the way the dimension affects the convergence curve 
is much less dramatic, corresponding to a simple scale dilation. This means 
that in some sense the complexity of the space Lq does not increase very 
much when the dimension increases. It is interesting to characterize, for 
several specific choices of G, the structure of Lq and to understand whether 
it contains a “sufficiently large” set of functions, where by “sufficiently large” 
we mean large enough to contain functions that are encountered in practical 
cases. This will be done in the next section for two particular choices of G. 


4 Examples of functions G 

In this section we consider two choices for the function G and study the 
corresponding functions spaces L G . We remind that for any given G £ L 2 (R d ) 
the space Lq is defined as 

L g = {/ £ L 2 (R d ) | / = G * A , A £ M(R d )} 

where M.(R d ) = M. is the space of Radon signed measures of bounded total 
variation on R d . 

4.1 The Gaussian 

We consider the Gaussian function G*(x) = e - H x H 2 , since approximation with 
Gaussian basis functions is often used in practical applications (Moody and 
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Darken, 1989; Poggio and Girosi, 1990; Poggio and Edelman, 1990; Sanner 
and Slotine, 1992). Clearly G £ Z 2 (-R d ), so that the space Lq is well defined 
in any dimension. Due to the smoothness of the Gaussian and to its fast 
decay property this space of functions is rather small. However it contains 
an interesting subset of the space of band limited functions, the functions 
whose Fourier transform has compact support. In particular, let us define 
the space of functions B k (R d )'. 

Bt(R d ) = {/ | / € Cl;(R d )} , (10) 

that is the set of functions whose Fourier transform has compact support and 
k continuous derivatives. Then the following inclusion holds: 

B k (R d ) C L G ,Vk> d -. (11) 

In fact if / £ B k {R d ) then we have 

fhi = ae"‘"7( S ) = A € , 

G(s) 

where a is a constant depending only on the dimension d. Therefore / = G* A 
where A is the Fourier transform of the function A = ^. Since the following 
inclusion holds (see appendix B): 

C^R 1 ) c A(R d ) - V(- > j . 

where A(R d ) is the space of the functions whose Fourier transform belongs 
to Li(R d ), then A £ L\ and / £ L G - 

We notice that the Gaussian function and its derivatives of any order 
belongs to X 2 , and therefore G £ H s ,2 for any s > 0. Hence we can apply 
corollary (3.1) to conclude that the convergence rate 0{-^=) also holds for 
approximation in the sup norm. 


4.2 Bessel-Macdonald Kernels 

We now consider the Bessel-Macdonald kernels, a family of functions G m (x) 
defined in terms of their Fourier transforms: 


Gm(s) 


1 

(1 + 47t 2 ||s|| 2 )t 


m > 0 . 
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The functions G m (x) are integrable functions that decay exponentially at 
infinity and may have a singularity at the origin (Stein, 1970, p. 132). How¬ 
ever if m > d they are continuous and actually differentiable of any order 
q < m — d. We want to work with continuous funtions and in what follows 
we will always make the assumption m > d. Since G m ( s) is positive and 
radial, we also have that, by Bochner’s theorem, G m (x) is positive definite 
(Micchelli, 1986), and therefore approximation by translates of G m (x) is a 
Radial Basis Functions approximation scheme. The following observations 
can be done regarding the functions G m and the space L Grn : 

1. One has 

G m G H s ’ 2 for 0 < s < m — — . 

Since we have made the assumption m > d one can take s such that 
| < s < m — |. Then we can apply corollary (3.1) to conclude that 
the rate of convergence 0(-^) also holds for approximation in the sup 
norm. 

2. Since R C M, the space L Gm contains the space jCl n (^R d ) = £} m of 
those functions that can be written as / = G m * A with A G R. For 
more information about the space which is a special instance of 
the so called potential spaces , the reader is referred to (Stein, 1970). 
The space £} m is related to the Sobolev space H m,1 (R d ) = H 171,1 of the 
functions whose weak derivatives up to order m are in L 1 (see Appendix 
B). More precisely one has (Stein 1970, p. 160): 

H™’ 1 C c} m C L Gm for all m even . 

Therefore we conclude that if m > d and m is even, by superposition of 
translates of G m we can approximate with a rate of convergence 0{-^=) 
all the functions of H 771 ’ 1 , and hence all C m functions which rapidly 
decrease to infinity. 

3. Again for s < m — | and m > d, m even, we have the following 
characterization of the space L Grn ■ 

L Gm = {fe H *’ 2 \ (i-A)?feM}. 
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In fact, if / G La m that is / = G m * A with A G M, then (/ —A)^/ = A 
since G m is the fundamental solution of the operator (I — A)~. On the 
other hand, if / G H S}2 and (/ — A)^/ = A G Af, then by taking the 
convolution of both sides with G m we have / = G m * A. 

5 Other Approximation Schemes 

Other choices of integral representation lead to different approximation schemes 
and different spaces of functions that can be approximated with a similar con¬ 
vergence rate. For example, using the Fourier representation of a function 
(if it exists) we have: 

/( x ) = / ds cos(s-x + 0(s))|/(s)| (12) 

JR d 

where 6( s) is the phase of the Fourier transform /(s) of /. Jones (1990) 
considers the space A(R d ) (appendix B) of the functions such that their 
Fourier transform is in Li(R d ) and shows that they can be approximated by 
functions of the form 


/n(x) = ^AiCOs(ti • x + 0i) (13) 

i = l 

with the rate of convergence 0(-^). 

Another result of this type has been proved by Barron (1991). He con¬ 
siders the set of the functions such that 

/ ds |M||/(s)| < +oo (14) 

JR d 

that is the functions whose gradient is in A(R d ), and approximates elements 
of this set by functions of the form 

n 

/n(x) = 'WA ' X + di) , 

8 = 1 

where cr(-) is any sigmoidal function. Condition eq. (14) can be rewritten as 

INIl/( s )| C Li(R d ). (15) 

Denoting by R the function 
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and noticing that its Fourier transform is R( s) = ||s|| _1 we can also say that 
the space of function that satisfy condition eq. (14) is the space of function 
that can be written as 


f = h* A , A £4(4 (16) 

There is a remarkable analogy between this set of function and the func¬ 
tion space £} m considered in section (4.2), that is the set of functions such 
that: 


/ = G m * A , A G Li(R d ) , m> d . (17) 

In eq. (16), the function Id goes to zero faster and faster as d increases, 
while its Fourier transform remains unchanged. In eq. (17), because of 
the constraint m > d, it is the Fourier transform of G m that goes to zero 
faster and faster as d increases, while the asymptotic decay of G m is always 
exponential. Moreover, in eq. (17) A has to belong to L\, while in eq. (16) 
it is the Fourier transform of A that belongs to L\. 


6 Conclusions 

We briefly summarize the main results presented in this paper. 

• Let / be a function on R d and assume that / can be written as / = G*\, 
where G is square integrable on R d and A is a signed Radon measure 
of bounded total variation. Then there is a linear superposition of n 
translates of G that approximates / in the L 2 norm with a rate of 
convergence 0{-^=). 

• Let / be a function on R d whose Fourier transform has compact sup¬ 
port and k continuous derivatives, with k > |. Then there exists a 
Gaussian Radial Basis Functions expansion with n basis functions that 
approximates / in the L 2 norm with a rate of convergence 0(-^). The 
same result holds for approximation in the sup norm. 

• Let / be any function of the Sobolev space H m,1 (R d ), with m > d, 
m even. Then there exists a Radial Basis Functions expansion, whose 
basis function is the Bessel-Macdonald kernel G* m (x), that approxi¬ 
mates / with a rate of convergence 0{-^=) in the norm of i7 s ’ 2 , with 
| < s < m — |. A similar rate of convergence can also be obtained for 
the approximation in the sup norm. 
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All these examples involve spaces of functions with a number of deriva¬ 
tives that increases with the dimension, and are consistent with the intuitive 
idea that spaces of function in a high number of dimensions are very difficult 
to approximate, unless some constraints are imposed to prevent their “size” 
to grow exponentially fast. 

One interesting feature of these results is that, thanks to the constructive 
nature of Jones’ and Barron’s lemma, an iterative procedure is provided that 
can achieve that rate. Clearly, these results concern the approximation of 
a function / which is known everywhere, while in many practical situations 
one would like to construct an approximation of a function / knowing only 
the values of / on some (finite) set of points. For this last problem, in the 
case of approximation by sigmoidal ridge functions, some results by Barron 
(1992) are already available, and show that also with this further source of 
error one can obtain results “independent on the dimension”, for suitable 
spaces of functions. It should be possible to obtain similar results for the 
approximation scheme we considered here, using the same technique. 

Acknowledgements We thank Tomaso Poggio for useful discussions and for a 
critical reading of the manuscript. 

A The Bochner Integral 

Let 0 C R d and let A be a positive measure on 0. For functions / : 0 — > X 
with X a Banach space there are several available notions of measurability 
and integration (Dunford and Schwartz, 1958; Diestel and Uhl, 1977). In 
particular for all (strongly) A-measurable functions / such that \\f\\x d\ < 
Too we can define the Bochner integral 

fdx . (18) 

Jn 

Clearly if A is a Borel measure the continuous functions / : 0 —> X are 

(strongly) measurable. One has lemma A.l below (Diestel and Uhl 1977, 
page 48). 

Lemma A.l Let X be a positive Borel measure on 0 C R d and /(t) : 0 —> X 
with X a Banach space. If f is Bochner integrable with respect to X then 

mlo 
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If one considers a signed Radon measure A on 0 one can still define the 
integral of a measurable function / : Q —)• X with respect to A as 


£ /(t)rfA(t) = £ (o) 

where |A| is the total variation of A and denotes the Radon-Nikodym 
derivative of A with respect to |A|. From lemma (A.l) one can easily obtain: 

Lemma A.2 Let X be a signed Radon measure on 0 C R d and /(t) : 0 — > X 
with X a Banach space. If f is X-measurable and is such that 

[ ll/ll <*|A| < +°° 

Jn 

then the Bochner integral of f with respect to X is well defined and 

jm£ mdm€7 ^- (20) 

where 


S = {sf(Ll) | s G R , |s| < 1} . 


In fact the scalar function ^yj(t) is measurable, the function /(t)^j(t) is 
measurable, and moreover 

/„'O'/o 11/11 d|A|<+ °°' 

Hence the integral f® f dX is well defined as the right member of (14). 
Then by lemma (A.l) applied to the function h(t) = /(t)^yy(t) one has: 

iam^ m M lt) d]m € “ hm ■ 

On the other hand since |4tt| = 1 one has 

I ci|A| I 

co h(fl) = co S 

and (20) follows. 
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B Sobolev Spaces and the Space A 

Here we collect a few facts about certain spaces of functions frequently used 
in the paper. 

Sobolev Spaces. For each positive integer s and 1 < p < oo one defines 
the Sobolev Space H s,p (R d ) = H s,p as the space of those L p functions in 
R d whose derivatives up to the order s are L p functions.The space H s,p is a 
Banach space with the norm 


E \\D°fh, 

\a\<s 

where a is a multi-index and D a is the derivative of order a. The space H s,2 
is a Hilbert space with respect to the scalar product 


i,v) = J2 / D<yu ° c 

I«I<S ' Rd 


One has also the characterization 


H s ’ 2 = {u e l 2 | (l + e l 2 } 

which can be used also to define the Sobolev spaces H s,2 for non integer s. 
One has the following result, which is a special case of the Sobolev embedding 
theorem (Stein, 1970): 

Theorem B.l If k is a positive integer and s > k + | then 

H s ’ 2 C C k 

and there is a constant c\ such that 


max sup \D a f(x)\ < ci||/|| ff ., 2 . 
\ a \< k xeR d 


The Fourier algebra A. The space A of the tempered distributions whose 
Fourier transform is a summable function is in current use in Fourier analysis 
(Herz, 1968; Katznelson, 1968). One has 

H k ’ 2 C A for k > ^ 

In fact (Barron, 1991; footnote) one may write 
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1 


/ = 


(i + M 2 )^ 


-[/(! + 


U)\ 


where both factors on the right side belong to U 2 if k > In particular it 
follows that C*q C H k ’ 2 C A for k > 

It is also clear that AcCo where Co is the completion in the L^ norm 
of Cq i.e. the space of continuous bounded functions that converge to zero 
for 11 x 11 — > oo. 
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